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Report Purpose 



The Regional Resource Center (RRC) Network Accountability Report documents the 
efforts and results of the RRC Network to: 

a) agree on a common set of performance measures; 

b) collect the data on those measures, by region; and 

c) aggregate those data and use them to make statements about the effects of 
RRC technical assistance for clients and their systems. 

This straightforward description understates a most significant achievement in the 
Network’s long history of attempts to demonstrate the extent to which RRC technical 
assistance makes a difference. 

The Report is organized by Performance Measure, followed by a collection of lessons 
learned, not only from the findings but also from the process of obtaining them. Ap- 
pendices offer the original protocols, and the full texts of the original Network Evalua- 
tion Framework and the Network Accountability Report Plan. An Executive Summary 
provides a succinct overview and highlights of the findings and lessons. 

It is important to stress that this effort focuses on the performance of RRCs, on their 
efforts and effects working with client states. In its RRFC Network partnership, the 
Federal Resource Center (FRC) provided substantial support, but as its mission, cli- 
entele and expected effects are distinct, no suggestion as to the FRC’s functions or 
performance are intended or should be inferred. Network, as used in this report, re- 
fers to the six RRCs. 
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Executive Summary 




Background 




The Regional Resource Center (RRC) program mission is to strengthen the capacity of 
state systems of education and early intervention to improve results for infants, tod- 
dlers, children and youth with disabilities and their families. It is one of the longest and 
most successful technical assistance projects of the Office of Special Education Pro- 
grams, US Department of Education. The full program currently includes six individual 
RRCs and one Federal Resource Center, but the focus of this effort is on RRC per- 
formance. Network, as used in this report, refers to the six RRCs. 

Despite continuous evaluation of efforts with clients and in regions, connecting client 
effects to improved child/student results has challenged the RRCs for years. Their 
2002 introduction to the Results Accountability framework (M. Friedman, Fiscal Policy 
Studies Institute) provided the most systematic approach to date to attempt tracking 
and connecting changes in student results to changes in the educational infrastructure 
attributable to RRC technical assistance. RRC Director approval of the RRFC Report- 
ing, Evaluation and Documentation (RED) Workgroup proposal launched the first pro- 
gram-wide effort to aggregate activities and effects of RRC technical assistance. 



The RRC Network agreed on four common performance measures: 

1) A “story” from each Center describing activities and effects to date 
in one state that reflect efforts in a priority area. 

2) Percent of states reguiarly impiementing, reviewing and revising 
improvement pians based on resuits data. 

3) Percent of clients reporting satisfaction on information requests. 

4) Percent of TAA evaluations that indicate TA was of high quality. 

The Network agreed to collect data on those measures by region, and then aggregate 
and report those data to make statements about the effects of RRC technical assis- 
tance for clients and their systems. Although the data the Network was able to collect 
did not always correspond to what was anticipated in the planning, the performance 
measures are presented as they were adopted, and experience became a lesson for 
planning the next round. 
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Findings 



Performance Measure 1: All Centers reported positive changes occurred as a 
result of the technical assistance provided. Evidence suggested that increased 
awareness among state-level and university-level policy-makers have led to im- 
proved state policies, such as changes in teacher certification standards or to im- 
proved practices in the areas of inclusive assessment, as well as data collection and 
analysis. 

Some Centers were able to report improved state data in the area corresponding to 
the technical assistance, such as increased rates of exit with a regular diploma and 
decreased dropout rates, increased rates of participation and achievement within 
inclusive assessment systems, and increased Part C identification rates. 

Performance Measure 2: RRC staff reported high percentages of states using 
results data to implement, review and revise improvement efforts, based on 
involvement with 37 Part B and 36 Part C agencies. Seven categories of 
improvement activities were weighted and displayed: self-assessment, improvement 
planning, focused monitoring, data systems, BPR/APR, GSEG, and SIG. 

A mean ranking of 6.4 (scaled 1-low, 8-high) was given by state clients to the 
connection between improvement planning and activities, and increased data-based 
decision making. State clients rated RRC assistance to their improvement processes 
very highly, with a 7.1 overall mean. 

Performance Measure 3: Percent of clients reporting satisfaction on information 
requests was high, a reported 4.4 mean (scaied 1-iow, 5-high) on an aggregate 
of 126 respondents. 

Performance Measure 4: Percent of TAA evaluations that indicate TA was of 
high quality was similarly positive, an overall mean of 7.0 (scaled 1-low, 8-high) 
on a Network aggregate of 479 evaluations. 
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Lessons Learned 



The challenges to mounting a first effort to establish, collect and report common 
performance measures across a national Network were significant but necessary to the 
RED Workgroup and the RRC Network’s commitment to program accountability. This 
ambitious effort yielded numerous lessons, expected and serendipitous. Measures, 
protocols and expectations underwent revisions as plans were put into place and ex- 
periences suggested modifications. 



The lessons that follow are a selection, those synthesized by the RED Workgroup 
members presenting the process and results in a session at the American Evaluation 
Association’s annual meeting in November, 2004. They are framed in statements of 
action that will contribute to the next stage in the RRC Program’s 
demonstration of its efforts and effects. 



Communication 

Clarify wording of Performance Measure (PM) questions 
Include representation from all interested entities in the redesign of instruments 
Build in periodic check-ins of a PM’s ability to collect appropriate data 
Communicate regarding performance measures throughout the process, between 
and among Centers 

Actions... 

Set expectations with clients regarding data collection 
Ensure that data collections are timely as well as collected at regular, 
agreed-upon intervals 

Ensure that we are asking the right questions and collecting the right data 
Distinguish data as useful to Centers and State clients 

Sometimes let data guide the performance measure, rather than the performance 
measure guide the data 

Examine staffing patterns for evaluation across all Centers 
Collaboration 

Establish expectations and common agreements among Centers 
Finalize a written Network evaluation plan 
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Introduction/Context 



Regional Resource Centers (RRCs) have endeavored since initial funding of the 
program in 1969 to document how they have performed and what difference that 
performance might have made for their clients. Ultimately, RRCs hoped their efforts 
would reach the intended beneficiaries of the system, children with disabilities and 
their families. In 1988, program grantees gathered to define and structure the nature 
of RRC technical assistance, and propose ways to evaluate and document that 
assistance. Figure 1 on the following page illustrates the growing attention and 
method brought to the puzzle by both RRCs and OSEP since that seminal event. 

Each period brought with it successes, challenges and increasing interest in getting 
better. The most recent RRC grant (1998 - 2003) occurred in the midst of heightened 
attention to accountability at all levels, and partnerships with states and OSEP in 
systematic improvement efforts offered an unprecedented opportunity to pursue 
individual Center and Network-wide efforts in performance measurement. 

In 2001, the Reporting, Evaluation and Documentation (RED) Workgroup of the 
Regional Resource and Federal Centers’ (RRFC) Network began building an 
evaluation framework to demonstrate the effects of RRC technical assistance on a 
program-wide basis. The RED group proposed to select common performance 
measures, collect data on them from each region, and aggregate the results and 
report them at the end of the RRCs’ most recent five-year grant cycle (originally. May 
2003). The RRC Directors reviewed and approved this proposal. The RED group 
agreed on four common performance measures and on an approach to collecting the 
data on these measures. The extension of the grants to May 2004 provided critical 
additional time for refining the approaches and collecting the data. 

This document reports the results of those efforts. 
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Figure I 

Regional Resource Centers’ Measuring 
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Findings 



Performance Measure 1 : 

RRC Efforts and Effects 

Performance Measure 2: 

State Use of Results Data 

Performance Measures 3 & 4; 
Client Feedback 
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PERFORMANCE MEASURE 1 
Stories of RRC Efforts & Effects 



In the course of collecting data on the measures, the RED Workgroup agreed to adjust 
the Network’s Performance Measure (PM) 1, 



Percent of states where the RRC intervened and positive trend 
lines are evident in one or more of the OSEP priority areas, 

to 

A “story” from each Center describing activities and effects to 
date in one state that reflect efforts in a priority area. 



This decision came with the realization that there likely would be insufficient data for 
each region (and therefore also the Network) to confidently report a percentage. 

Priority areas targeted by the Centers included Exiting, inciusive Assessment, 
Disproportionality, and Part C Identification. To address these areas of State need, 
RRCs partnered with OSEP, an array of TA&D Centers and other interested entities in 
developing and implementing technical assistance plans. These partners included: 
the National Center for Educational Outcomes (NCEO), Interstate New Teacher 
Accreditation Standards Consortium (INTASC), Center for Positive Behavior 
Intervention and Supports (CPBIS), National Early Childhood Technical Assistance 
Center (NECTAC), Department of Children and Families, the African and Caribbean 
American Families of Children with Disabilities, the NAACP, parent advocacy centers, 
state and local departments and boards of education, education associations, and 
multiple colleges and universities. 

All Centers reported positive changes occurred as a result of the technical 
assistance provided. Centers offer evidence that increased awareness among 
state-level and university-level policy-makers have led to improved state policies, such 
as changes in teacher certification standards or to improved practices in the areas of 
inclusive assessment, as well as data collection and analysis. 

Some Centers were able to report improved state data in the area corresponding to the 
technical assistance, such as increased rates of exit with a regular diploma and 
decreased dropout rates, increased rates of participation and achievement within 
inclusive assessment systems, and increased Part C identification rates. 

These stories offer a modified national “baseline” of data collection and analysis, and 
will contribute to continuing Network efforts to obtain and report reliable data at both 
individual Center and program levels. The Network has long attempted to effectively 
demonstrate the results of its technical assistance while recognizing that the clients 
(states) actually hold the ultimate data (Figure 1). These stories comprise the first 
collective endeavor, in what is anticipated to be an ongoing effort, to align RRC 
technical assistance with those state data. 



- Page 8 - 



Region 1: Northeast Regional Resource Center (NERRC) 
Involvement with Connecticut 
Disproportionality Activities 




state Context: 




Connecticut (CT) identified concerns about overrepresentation of minorities in Special 
Education in several of its larger, urban centers. To assess and help address some of 
these issues, the CT SEA representatives requested technical assistance from the 
NERRC. 

In October of 2002, NERRC held a two-day meeting in Springfield, MA, entitled 
“Addressing Disproportionality in Special Education,” which had a significant impact on 
CT’s efforts to address the issue. Director George Dowaliby, Bureau Chief for Special 
Education and Pupil Services for the CT SEA, led a team that included Dr. Nancy 
Cappello, a Bureau consultant. The team used the conference to begin planning for a 
statewide summit it was holding in March 2003 to address disproportionality. Dr. 
Cappello was the lead contact person for CT on the project. NERRC participated on 
the team that planned the summit. 

Following the NERRC-sponsored conference in MA, CT identified 34 LEAs that had 
serious issues with disproportionality, based upon district data, and invited them to 
send teams to participate in the summit. As part of their work at the summit, the 34 
district teams drafted action plans about how they would tackle disproportionality. An 
analysis showed many common strategies or activities that districts identified: 
reviewing and using data effectively, providing staff training, improving instructional 
practices in general education (e.g., implementing differentiated instruction), and 
providing professional development on cultural competence. 

CT invited three of the speakers from NERRC’s MA conference - Drs. Gwen Webb- 
Johnson, Margaret McLaughlin and Darren Woodruff -to participate in the March 2003 
summit. Dr. Webb-Johnson was the keynote speaker for the first day of the CT summit 
and also conducted a workshop. Drs. McLaughlin and Woodruff conducted workshops 
and served on a panel. NERRC sponsored Dr. Woodruff’s participation. NERRC also 
researched other participants for the summit, including Dr. Eric Smith, superintendent 
of schools for Anne Arundel County, MD, who was the keynote speaker for the second 
day of the event. 

In May 2003, the CT SEA invited the local district teams back for a follow-up meeting. 
NERRC researched the keynote speaker for the event. Dr. Ronald Ferguson of 
Harvard University. Twenty-two districts participated. Each district received a $1 ,500 
planning grant and 1 1 of the districts received $5,000 grants to implement their plans. 
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As part of the summit work, two state level teams, one on Policy and Governance and 
the other on Teacher Preparation, met and developed a series of recommendations to 
address disproportionality and the achievement gap. NERRC participated on the State 
Policy and Governance Team that drafted recommendations on how to close the 
achievement gap and address disproportionality and over-identification. 

Evaluations gave high marks for both the March and May 2003 events. In May, almost 
two of three participants reported they were doing things differently as a result of the 
summit. Action steps being taken included: analyzing data, sharing information with 
colleagues, assembling committees, reviewing assessment tools, offering 
professional development and forging new alliances between special and general 
education. 

Summit II was held in March 2004. NERRC identified and recommended the keynote 
speaker. Dr. Pedro Noguera of New York University, formerly of Harvard University, 
and he was given high marks for his presentations. Twenty-nine districts were 
represented and 255 people attended. Participants said it was a success, with half 
calling it “excellent” and 41 percent saying it was “very good.” Eighty percent said they 
believed a third summit is needed to keep repeating the message and to monitor 
progress. 




NERRC’s Technical Assistance: 




• Our October 2002 conference gave CT information they could use in planning 
their summits, but also informed how they think about disproportionality. CT 
SEA staff also made contacts with several speakers and presenters they invited 
to present at their summit. 

• NERRC researched and made recommendations on other participants in CT 
summits, including the keynote for Summit I, the keynote for the May 2003 
follow up and one of the two keynote speakers and presenters at the Summit II 
in March 2004. 

• NERRC participated on the stakeholders’ group that planned the summits and 
the Policy and Governance State Team which made a series of 
recommendations to the education commissioner’s office on how to address 
disproportionality and the achievement gap. 

• NERRC regularly supplied the SEA with important articles and resources that 
address disproportionality or the achievement gap. 

In addition to NERRC, some of the partners involved with the DOE in this effort are: 
the Department of Children and Families, the African and Caribbean American 
Families of Children with Disabilities, the CT Association of Boards of Education, the 
State Advisory Council, the CT Parent Advocacy Center, the Commission on Children, 
the CT Education Association, the NAACP, and several colleges and universities. 
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Results to Date: 



According to the DOE contact leading the disproportionality effort, although there is as 
yet no “hard” data to show activities are working, there is a demonstrable change in 
education officials, who now understand the issue much better and realize they can 
do things to reduce disproportionality and close the achievement gap. 



The CT Data Manager reported that the formula they used to assess disproportionality 
changed between the two years so they are not equivalent groups, thus it is not 
possible to attribute improvement at this time. She continued, ‘We used 2001- 2002 
data to invite districts to the summit in March of 2003. We then used 2002- 2003 data 
to invite districts to the summit in March of 2004. But, the year two anaiysis data, 
2002- 2003 data, were already collected (Dec. 2002) before we held the first summit in 
March of 2003. Impact from the Summits should be seen in the Dec. 2003 data that is 
being ‘cleaned’ right now. ” 



NERRC has contributed to the dramatic change in mindset in the SEA about 
disproportionality. Many special educators admitted earlier that they did not see the 
importance of disproportionality, or they believed it was only an economic issue they 
could not influence. Now, there is a focus from the commissioner’s office to the school 
building level that disproportionality is an important issue and there is much that can 
and must be done about it. 
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Region 2: Mid-South Regional Resource Center (ASC/MSRRC) 

Delaware’s Assessment System 



state Context: 




The Delaware (DE) Department of Education tried for several years to move toward a 
fully inclusive large-scale assessment system. Among their initiatives were a research 
project funded through the U. S. Office of Educational Research and Improvement on 
accommodations to the Delaware Student Testing Program (DSTP), development of 
an alternate assessment approach using a portfolio, and extensive work on the state 
accountability formulas to better include students with disabilities and students who 
have English as a second language. However, the data indicated that a significant 
number of students with disabilities were still exempt from the DSTP. There was a 
need to broaden the range of acceptable accommodations, complete and implement 
the alternate portfolio efforts and integrate all students with disabilities in the 
accountability formulae. As test data became available and the DE database became 
more sophisticated (in 2001-2002), new questions arose regarding such issues as: 

1 . The extent to which students who were scoring poorly were poorly instructed or 

were inappropriately tested, 

2. The relationship between placement and test scores, 

3. The extent to which students were being successful over time (longitudinal data), 

and 

4. The extent to which there was bias in the test items. 




ASC/MSRRC’s Technical Assistance: 




ASC/MSRRC assisted Delaware in moving toward a fully inclusive large scale 
assessment system by providing information and consultation to SEA staff about policy 
and state practices, by advising the state task force on accommodations, by serving on 
the alternate assessment advisory committee and general Technical Advisory 
Committee (TAC) for the Delaware Student Testing Program (DSTP), by providing 
consultation on question writing and data analysis and by assisting them with an 
analysis of data on students who are scoring in less-than adequate ways on the DSTP. 
The RRC assistance was in part a collaborative effort with Martha Thurlow from the 
National Center for Educational Outcomes (NCEO) and other assessment 
stakeholders. 
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Results to Date: 



TAG participants and Delaware state staff were interviewed regarding the impact of 
ASC/MSRRC assistance on the state testing program. Each interviewee mentioned 
that ASC/MSRRC had been integral in assisting the state make connections with other 
states, NCEO, TAC experts from around the nation, and introducing them to the Tindal 
and Fuchs article on assessment and instructional accommodations, an article ASC/ 
MSRRC contracted to write. The interviewees noted an impact as a result of ASC/ 
MSRRC work. A Delaware SEA staff member declared, “All kids are included in the 
alternate [assessment] or the regular assessment with accommodations. We wouldn’t 
be there without MSRRC help.” Other participants made comments such as: “With 
[ASC/MSRRC] help, Delaware is one of the first groups to meet federal assessment 
requirements.” Overall, Mid-South’s assistance has been shown to be effective in 
informing the correct state staff of policy issues and assisting in achieving policy 
change that brings Delaware closer to full compliance with many federal education 
requirements. Its influence has stretched to the classroom level since students have 
access to a variety of accommodations for both instruction and assessment that were 
not available prior to 1999. 



The following charts show the percent of special education students meeting the DSTP 
content standards for reading, math, and writing. The number of students with 
disabilities participating in the large scale assessments has increased from 2001 to 
2003. Concurrently, the scores for students with disabilities have generally increased 
during this time period in each of the grades tested, as the trend graphs below depict. 
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Percent of Delaware Special Education Students 
Meeting/Exceeding Standards in Writing, 2001-2003 




• Grade 3 

• Grade 5 

• Grade 8 

• Grade 1 0 
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Region 3: Southeast Regional Resource Center (SERRC) 
SERRC, INTASC, CPBIS, and the Alabama Teacher 
Education Standards Alignment Project 



state Context: 




The State of Alabama was one of the first states to implement an exit exam as part of 
the requirements for students to receive a regular diploma. Since its inception in the 
early ‘80s, the exam has been revisited on a regular basis to confirm that its difficulty 
ensures that students are well prepared to meet workforce and college preparatory 
demands, as well as the necessary Carnegie units required for successful high school 
completion. The present graduation requirements were in place for the graduating 
class of 2001 and included four years of electives and higher-level English, math, 
science, and social studies (also referred to as the 4 x 4 curriculum), thereby increas- 
ing the required number of credits from 21 to 24. The graduation requirements also 
increased the examination’s difficulty from the previous eighth grade level to eleventh 
grade level, and science and social studies exams were phased in during subsequent 
years. 

Alabama’s high school graduation requirements were - and still remain - among the 
nation’s highest. However, the increased rigor was not without its cost. Prior to the 
increase in the exam’s difficulty, just over 30% of students with disabilities had been 
able to pass the exam and receive a regular high school diploma. Many of those 
students had historically been taught basic subject matter by special education 
teachers (e.g., general math) who did not necessarily hold certification in the subject 
area taught, but those teachers had still been able to assist the students to pass the 
exam. However, state officials could see that the new, more rigorous academic 
demands could only be met by teachers with appropriate content area expertise - 
usually, the regular education teacher. 

Although Alabama had previously made progress toward including more students in 
general education classes, many teachers cited their lack of expertise in providing 
appropriate accommodations to assist students with disabilities in regular classes. / 
didn’t go to school and get prepared to teach special education students! teachers 
often said. Since the state education budget was dependent upon sales tax revenue, 
recent economic downturns had resulted in budget cuts in local education agencies, so 
additional personnel for co-teaching was, at best, mere wishful thinking for local 
superintendents. A systemic intervention aimed at the “root cause” was necessary to 
achieve the level needed to pass the new graduation exam and the required rigorous 
coursework. Clearly, the confluence of the requirements of the No Child Left Behind 
Act, the demands of the modern workforce, and greatly increased graduation 
standards meant that all teachers needed to be prepared to teach all students in their 
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least restrictive environment, if increased numbers of students with disabilities were to 
receive a regular high school diploma. Improved post-secondary outcomes for 
students with disabilities depended upon the students’ receipt of a regular diploma, as 
post-secondary options were limited for recipients of the Alabama Occupational 
Diploma and graduation certificates. 



SERRC’s Technical Assistance: 



The Alabama state director of special education and members of his staff, the Alabama 
state director of teacher certification, a dean of a school of education, and the 
Southeast Regional Resource Center (SERRC) director and staff determined that, 
without intervention, the graduation rate would gradually improve; however, the rate of 
improvement or of turning the curve, was not adequate. SERRC and the Alabama 
directors met with the INTASC staff to design a meeting composed of a variety of 
stakeholders to align Alabama’s teacher certification standards with those of CEC and 
the Interstate New Teacher Accreditation Standards Consortium (INTASC), a TA&D 
funded entity. The initial meetings were facilitated by SERRC in partnership with 
INTASC in November and December 2002, with follow up meetings held in 2003. The 
stakeholders present at the meetings included deans of all Alabama state schools of 
education, higher education faculty, general and special education teachers, business 
representatives, parents, and concerned citizens. 





Results to Date: 




Although it may be many years before the promise of this strong innovation is evident 
in student data, the meetings have thus far shaped new policy directions. The revised 
teacher certification standards have been approved by the state board of education 
and have been codified. Students in all state schools of education receive a copy of 
the Knowledge, Skills, and Abilities (KSAs) they are expected to demonstrate before 
being certified as teachers in the state of Alabama. Education coursework is 
structured based upon the KSAs and teachers design tests accordingly. Students who 
fail to meet the standards are counseled by their instructor and meet to develop 
improvement plans. 

In addition to changes in policy and expectations for teacher education students, a 
clustering of professional development opportunities has been offered to Alabama 
teachers in a wide range of venues. For example, teachers have access to electronic 
professional development CDs that offer information on their desktops regarding 
accommodations for students. The Alabama State Improvement Grant (SIG), in 
collaboration with SERRC, has been vigorous in its efforts to provide teachers with 
scientifically based strategies to improve teaching and learning for a wide range of 
students. Partnerships have been strong with the OSEP-funded Center for Positive 
Behavior Interventions and Supports (PBIS) to implement training for schools to 
improve student behavior. Moreover, th^^gj^^n increased awareness on the part of 



the education community from higher-education faculty to classroom teachers that all 
teachers are expected to teach all students. This expectation is creating positive 
results, including changes in state board regulations that now extend waivers to 
students with documented disabilities who fail graduation exam sub-tests while still en- 
couraging students to work toward high standards in the other subject areas. 

Julia Causey, Alabama State Department of Education; We had a difficult 
task which SERRC made possible and feasible through their planning. As a 
result of this technical assistance from SERRC, we went to the State Board of 
Education with the recommendations from our committee’s work and made 
changes in the SDE guidelines. We have revised our certification procedures 
at the SDE and are now planning next steps - long lasting results. 

Although available data indicate low numbers of students with disabilities (as well as of 
students without disabilities) exiting high school with a regular diploma, Alabama has 
implemented a number of initiatives designed to increase the high school 
graduation rate, including teacher pre-service and inservice training, as well as 
exploring additional options to increase the avenues available for students to achieve 
graduation with a regular diploma. The table below shows graduation rates for 
students with disabilities receiving the regular diploma and the Alabama Occupational 
Diploma. 



Year 


Graduation Rate for SWD 


Diploma Rate for 
AOD 


98-99 


31.6 


n/a 


00-01 


22.0 


n/a 


01 -02 


22.5 


22.3 


02-03 


17.4 


21.4 



SWD = Students with disabilities 
AOD = Alabama Occupational Diploma 
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Region 4: Great Lakes Area Regional Resource Center (GLARRC) 

Complaint Investigator Training 



state Context: 




Driven by requests from State Directors of Special Education, GLARRC contracted 
with Kevin C. McDowell, J.D., General Counsel for the Indiana Department of 
Education, to train new and experienced State Complaint Investigators and others on 
conducting investigations and writing reports. 




GLARRC’s Technical Assistance: 




The Complaint Investigator Training course addressed the following four objectives: 

• To introduce a model framework for conducting legally sufficient complaint 
investigations. 

• To identify the steps which a complaint investigator must follow and the decisions 
to be made when conducting an investigation. 

• To introduce cognitive and procedural strategies which are used by experts in the 
field. 

• To provide a forum to discuss ideas, techniques, issues, and problems relating to 
the investigation of complaints. 



Five Complaint Investigation Training courses were held. State Complaint Investiga- 
tors and Regional Resource Center staff members from across the nation were invited; 
a total of 129 representatives from 38 states attended. Table 1 displays participants by 
state for each of the five training sessions (RRC participants are not included in this 
table). Massachusetts (n=17), Illinois (n=12), New York (n=9), Pennsylvania (n=9), 
Maryland (n=8) and West Virginia (n=8) had the highest state representation at the five 
training sessions. 
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Table 1 : State Participants in the Compiaint investigation Training Sessions 



state 


6/00 

CO 


8/00 

MO 


7/01 

MA 


7/01 

IL 


7/02 

MO 


Arizona 


1 


3 








Bureau of Indian Affairs 


1 










Colorado 


2 










Connecticut 


1 




2 






Delaware 








1 




Kansas 


1 


2 








Hawaii 










2 


Idaho 




1 








Illinois 




1 




9 


2 


Indiana 










2 


Iowa 


1 










Kentucky 








1 




Louisiana 








2 




Maine 






1 






Maryland 




3 




5 




Massachusetts 






17 






Michigan 


3 








2 


Minnesota 


3 










Missouri 




2 






1 


Montana 


1 










Nebraska 


1 










New Mexico 


3 










New Jersey 




1 


2 






New York 


1 




8 






North Carolina 


1 


1 








South Carolina 








1 




Ohio 


1 


3 






1 


Pennsylvania 


2 






5 


2 


North Dakota 


1 










South Dakota 


1 










Texas 




1 








Utah 


1 










Vermont 




1 


2 






Virginia 








1 


2 


Virgin Islands 










3 


West Virginia 


2 


3 




3 




Wisconsin 










1 


Wyoming 


1 






















Total = 38 states 


29 


22 


32 


28 


18 



Methods 

Data were collected through a participant survey, sent to 89 State Complaint 
Investigators who participated in the five training sessions. Twenty-one participants 
returned the survey for a response rate of 24%. The survey asked about the impact of 
the Complaint Investigator Training: 

1 . In what ways did State Complaint Investigators use the information presented at 
the training? 
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2. Did participation in the Complaint Investigator Training help facilitate change at 
the state and district levels? 

3. Are the complaint investigative procedures implemented in the states meeting 
the needs of students with disabilities and their families? 




Results 




Table 2 responses indicate that the information was used predominantly for investigat- 
ing and writing complaints; 20% used the information in providing consultation/ 
technical assistance services. 

Table 2: Participant Responses identifying Activities For Which the information 
Was Used 



Activities 


Tot 

% 


6/00 

CO 


8/00 

MO 


7/01 

MA 


7/01 

IL 


7/02 

MO 


Providing consultation/technical assistance ser- 
vices 


20% 


3 


3 


0 


1 


7 


Planning and conducting training 


4% 


1 


1 


0 


0 


2 


Developing & disseminating print information 


10% 


2 


3 


0 


0 


2 


Investigating complaints 


29% 


4 


6 


2 


1 


7 


Writing Complaints 


28% 


4 


5 


2 


0 


7 


Others: 


9% 


1 


1 


0 


0 


4 



The participant survey also asked participants in what types of tasks they used the in- 
formation associated with Complaint Investigations. Table 3 illustrates the array and 
percentages from the respondents. 



Table 3: Participant Responses identifying the Compiaint investigations Tasks 
For Which the Information was Used 



Activities 


Tot 

% 


6/00 

CO 


8/00 

MO 


7/01 

MA 


7/01 

IL 


7/02 

MO 


Determining the validity of complaint allegations 


14% 


3 


4 


2 


1 


7 


Determining sufficiency of findings of fact 


14% 


3 


5 


2 


1 


7 


Determining the need for additional supporting 
documentation 


13% 


2 


4 


2 


1 


7 


Defining solutions to the presenting problem 


10% 


3 


2 


1 


0 


6 


Determining corrective actions 


10% 


3 


3 


1 


0 


5 


Writing formal complaints 


10% 


4 


3 


1 


0 


4 


Determining corrective actions 


10% 


3 


3 


1 


0 


5 


Writing letters of findings 


13% 


3 


4 


2 


2 


5 


Making legal decisions 


10% 


1 


2 


1 


1 


5 
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Seventeen respondents indicated their participation in the training helped facilitate a 
change in their states; four reported no change. Table 4 summarizes the participant 
responses to ways that training impacted on or resulted in a change at the state level. 
Table 4 illustrates that 13 (24%) of the responses indicated that investigative 
procedures and report writing changed at the state level, while 12 (22%) responses 
indicated evaluation of complaints and nine (17%) indicated writing complaints 
changed at the state level. 



Table 4: Participant Reports of Training Impacts at the State Level 





n 


% 


Investigation procedures changed/refined 


13 


24% 


Review of investigations changed 


7 


13% 


Writing complaints changes 


9 


17% 


Report writing changed 


13 


24% 


Evaluation of complaints changes 


12 


22% 


Table 5 displays responses to the same question about impacts at the district level. 
Eight respondents reported change at the district level in their states, while 1 1 respon- 
dents indicated no change. 


Table 5: Participant Reports of Training Impacts at the District Level 






n 


% 


Investigation procedures changed/refined 


3 


23% 


Review of investigations changed 


3 


23% 


Writing complaints changes 


3 


23% 


Report writing changed 


3 


23% 


Evaluation of complaints changes 


1 


8% 



Changes for Students and Families 

Participants were asked, “Do you think the complaint investigative procedures that are 
implemented in your state are meeting the needs of students with disabilities and their 
families?” Of the 21 responses, 18 (86%) indicated yes while 3 (14%) indicated no. 

Participants also responded to an open-ended question on how complaint investigative 
procedures are meeting the needs of students with disabilities and their families. Indi- 
vidual 

responses included: 

V The process complies with the law. 
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V Complaint procedures have been streamlined. 

V An appeal process helps ensure correct results. 

V Parents are not shy about using the system. 

a/ Parents know they will get immediate action. 

V There is a timely resolution of complaints. 

a/ Parents are provided with various alternatives. 

a/ State Departments of Education are rarely contacted regarding final reports and 
decisions. 

a/ Parties feel that the process is fair and impartial. 

a/ a statewide system of parent advocates was developed using state and federal 
funding. 

a/ Advocates are trained in special education law. 

a/ Information provided to school officials and parents is clear. 

Participants identified the following needed improvements: 

a/ Principals need to be held accountable for decisions made at the building level. 

a/ The process needs to focus more on the provision of PAPE and achievement for 
the student, not just the specific complaint allegations. 

a/ The complaint regulations do not address relationship issues the ways media- 
tions can. 




The Complaint Investigator Training provided state personnel with information used to 
investigate complaints, write complaints, provide consultation/technical assistance ser- 
vices, and develop and disseminate print information. Participants reported that the 
information was used in determining the validity of complaint allegations, sufficiency of 
findings of fact, and the need for additional supporting documentation. Participants 
also indicated that the information helped them to define solutions to the presenting 
problems, determine corrective actions, and write formal complaints, and reported that 
their participation in the training helped facilitate changes at the state level. Respon- 
dents reported specific changes that included revision or refinement of complaint pro- 
cedures, process for complaints review and writing, and how reports are written. Re- 
spondents to the impact evaluation indicated that the investigative procedures being 
implemented in their states are meeting the needs of students with disabilities and their 
families. 

Other unsolicited impact of the training included the following comments made by state 
staff: 

• 7 thought the training was excellent at the time, but it’s in the reflecting on the 
training during our practice that I’ve realized how really great the training was. ” 

• “I liked having time with people from other states doing the same thing as I do. ” 

• “We enjoyed the training conducted by Kevin McDowell. By the way, we had 
him conduct training for our region (sponsored by MPRRC) in June 2002. ” 



- Page 22 - 



z; 



Region 5: Mountain Plains Regional Resource Center (MPRRC) 

Transition in Montana 




state Context: 




Montana, upon completing a self-assessment and OSEP visit as part of the 
Continuous Improvement Monitoring Process (CIMP), recognized that despite almost 
10 years that included a federal transition grant and considerable training, secondary 
transition was not occurring as it was intended. It was obvious that the IDEA 
regulatory requirements for transition planning were still missing from many student 
files. While MT realized that focusing on regulatory requirements did not automatically 
lead to better outcomes for students, MT staff did realize the importance of quality 
transition planning for improved student outcomes. They realized that many lEP teams 
did not understand “what” or “how” to implement the transition requirements and thus 
did not have the foundation needed for effective transition planning. 




MPRRC’s Technical Assistance: 




Montana learned about a pilot project operating in Wyoming under the leadership of 
MPRRC. This pilot project, which has become known as TOPs (Transition Outcomes 
Project), was demonstrating some impressive preliminary results in pilot districts. 
Montana expressed an interest in replicating this project and requested assistance 
from MPRRC. 

MPRRC met with representatives from the Montana Office of Public Instruction and a 
preliminary plan was developed to pilot the TOPs project. Initially, a video conference 
was held across the state to explain the project to districts. Following the video 
conference, two districts volunteered and within three months an additional 53 districts 
had volunteered. While improved post-school outcomes was the long range desired 
result, it was clear that the first step was to increase the documentation that transition 
planning was taking place in line with the IDEA regulatory requirements. To reach the 
immediate result of improved transition planning, the following activities were identified 
and implemented: 

• Refine evaluation instruments and lEP review procedures. 

• Identify and train evaluation team members on the instrument, process, and 
procedures. 

• Select districts and review team members, and visit districts for lEP reviews. 

• Compile, analyze, and summarize findings. 
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• Follow up in local districts to report findings from the lEP review; develop 
strategies; and set target goals, timelines, and schedules for addressing the 
transition requirements that were found to be a problem. 

• Provide training to local education agency personnel and associated staff. 

• Provide follow-along advice and training as local education agencies implement 
the skills, knowledge, strategies, and interventions. 

• Conduct a final review of files to determine if change had occurred. 

These activities were based upon the TOPs conceptual framework and belief that if 
one could 

• determine the specific problems in developing and implementing each of the 
transition requirements, 

• work with teachers and have them help identify and develop strategies to try to 
resolve the problems at the level where the problems existed (in schools and 
buildings, with lEP teams), and 

• provide concrete examples and strategies for possible changes that could be 
put into place that demonstrate improvement, 

then the requirements could be met, improvement demonstrated, and results shown. 




Results to Date: 




Baseline data using the TOPs Checklist was gathered on over 1 ,000 students 14 years 
of age and older from across Montana during the 2000/2001 school year. One year 
later, during the 2001/2002 school year, final data were gathered on students from the 
same districts to determine any change regarding the transition requirements. Below 
is a summary of the baseline and final data. 



Requirement 


Baseline^ 


Final, 


Student Invitation 


67% 


95% 


Invitation of other agency 


27% 


29% 


Statement of Transition Service Needs 


46% 


65% 


Statement of Needed Transition Services 


74% 


91% 


SNTS - coordinated set of activities 


38% 


43% 


SNTS - activities promote movement to post school 


46% 


61% 


Statement of interagency responsibilities/linkages 


17% 


47% 



While none of the transition requirements met 100% compliance, it was clear that 
they had improved. In addition to the training conducted with review teams, specific 
training on each of the requirements was a large part of the Report Out meetings in lo- 
cal districts. The Report Out Meeting process in the TOPs project recognized several 
key aspects to change with staff in local districts. 

1 . Staff in each district was presented with the baseline data with the 

understanding that the information had nothing to do with compliance. It was 

^ Baseline n = approximately 1 ,100 lEPs of students 14 years of age and older 
^ Final n= approximately 1 ,000 lEPs of students 14 years of age and older 
®Time period from baseline to final was 12 months. 
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explained that if they choose to do something different they first must know and 
understand what they are doing and how well - a function of the baseline data. 

2. The TOPS model recognizes that if change is going to occur with staff that is 
primarily responsible for the requirements (i.e., special education teachers) they 
must “own the problem.” Staff was presented with the option to set whatever 
goal they desired on each of the requirement findings. If their goal was greater 
than the baseline data results then they - the staff - “owned the problem.” 

3. The TOPS model also recognized that the staff must “own the solution” to the 
problem. As a part of the Report Out Meeting process, staff was asked to 
identify solutions in order to achieve their desired goal. It was believed that the 
teachers, once they understood the requirement and where they were currently 
functioning, could identify solutions or strategies in order to improve. 

4. Another critical component to the Report Out Meeting process was to allow the 
staff to set a timeline for making the necessary changes with the understanding 
that at the designated future date the review team would come back to review 
files once again. Most staff stated that they needed one full year or one full lEP 
cycle to make the changes. 
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Dropout Rates for Students with Disabilities = Dropouts divided by Total Students with disabilities, ages 14-22+. 

In some cases the dropout count numbers are relatively low causing a wide variation in the dropout rates. This variation may 
suggest a discrepancy where, in fact, the numbers are too small to be statistically significant. 





Reports from the clients: 

My staff told me that they have sent you some information. I hope that someone 
passed along that we made our new Transition [Outcomes Project] goals last year with 
100 percent of our students who were eligible fora driver’s license graduating with 
one. [Also], 100 percent of our students graduated from high school with an unpaid or 
paid work experience or two! 

The Outcomes Project has given renewed meaning for special education staff. It gave 
importance and relevance to what they were doing with and for kids. It changed the 
focus of our programs at Ronan Middle and High Schools. Students understand that 
the choices they make today will affect them for future employment. We have become 
acquainted with our local service providers and held our first transition fair last spring 
with great attendance. 

— Joan Graham, Director of Special Education, Ronan, MT 

The MTOP [Montana Transition Outcomes Project] has served to “keep the Transition 
fires burning” in Montana. The project has become the foundation to build on for 
improved Transition activities. The benefits include 

• Development of a cadre of people who have knowledge and expertise and are 
now poised to provide technical support and training to others in the State; 

• A forum to develop more consistent practices and a greater understanding of 
Transition, statewide; 

• Allowing OPI to provide leadership to educators and develop guidelines; 

• Improved student/parent invitation forms to reflect federal transition 
requirements; 

• Established systematic approach to review schools’ lEP forms regarding 
compliance with federal guidelines; 

• Established systematic approach to provide technical assistance to schools 
ensuring ongoing improvement; and 

• Improved agency/school interaction and dialogue resulting in better transition 
outcomes for students. 

— Mike Peterson, Director, Montana Center on Disabilities, MSU, Billings 
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Region 6: Western Regional Resource Center (WRRC) 
Process to Performance: Nevada Part C Turns 

the Curve 



state Context: 




In Summer 2000, NV Part C staff attended the Self-Assessment Institute co-sponsored 
by the RRFC Network, NECTAC and OSEP. In July 2001 , the WRRC sponsored a 
meeting for Alaska, Oregon, and Nevada, those states in the Western region about to 
embark on their first self-assessment (SA) process. The meeting focused on 
expectations and resources, and featured the SEA Director from Idaho describing their 
just-completed experience. Although NV Part C was not at this meeting, staff had 
separately contacted the WRRC for help with its Continuous Improvement Monitoring 
Process. NV Part C staff attended the 2001 Summer Institutes on Self Assessment (SA) 
and Implementation Planning (IP), again jointly offered by the RRFC Network, NECTAC 
and OSEP. 




WRRC’s Technical Assistance: 




NV Parts B and C formed a Joint Coordinating Committee for their SA, but maintained 
parallel processes, with WRRC support to both. Each Part had its own manager, and a 
representative from “the other” Part on its Steering Committee, and used the WRRC for 
information and facilitation and NECTAC assistance during the process. 

In December 2001 , NV Part C submitted its final Self Assessment to OSEP. In 
January 2002, the WRRC held a seminar on Results Accountability (RA), intended to 
support the region’s states in both their SA and IP processes. NV Part C asked the 
WRRC to introduce RA to its Steering Committee during their Improvement Planning 
orientation meeting in March. The RA tools were adopted to facilitate and complete the 
IP process, for which the WRRC provided continuous facilitation. 

Concomitantly with the IP process, NV Part C was being relocated from the 
Department of Human Resources to the Department of Health. The SA process had 
provided “the state” a public forum for informing and engaging stakeholders to look hard 
at the major noncompliance areas (reported in their SA document). During this same 
period, an internal audit had revealed inconsistencies in the Part C program; that report 
suggested part of the difficulty was flat funding from the NV Legislature for the past 
seven years. Part C staff became much more public with their frustrations and the lack 
of progress. Being transparent was the best thing we ever did, reported Wendy 
Whipple, the Part C Coordinator. Taking OSEP’s offer of amnesty for honesty at face 
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value, NV Part C staff purposefully used the SA and IP processes to promote 
significant changes in services to infants and toddlers. When OSEP made its 
verification visit in July 2003, and asked “the state” (now the Department of Health) 
whether they were interested in fixing their issues, the answer was a resounding “yes.” 
The shift in scrutiny and in lead agency highlighted the variations in service delivery 
models in the state. This, coupled with the growing recognition of the need for data, 
fostered a re-examination of the overall philosophy and mission of Part C. Stemming 
from this came a renewed team-based approach to family-centered services in 
natural environments. Families were increasingly involved in choices and decisions. 
The trend in “drop-outs” from early intervention began to turn. [NV Part C presentation 
at WRRC Forum on Focused Monitoring, 2003] 

Once NV Part C completed its Self-Improvement Plan in July 2002, agency staff set 
about implementing its improvement strategies. The agency again asked NECTAC and 
WRRC staff to continue their resource and facilitation partnership for the various 
committees charged with priority activities. The involvement and resources from these 
technical assistance sources promoted stakeholder and provider considerations of 
different models; NECTAC’s guidance and the WRRC’s support in helping NV Part C 
“re-engineer its services” have resulted in significant improvements in the service 
system. There were also changes in community collaboration and a focus on best 
practices for providers, and a concerted effort to systematize the new approaches and 
practices statewide, using data to target where needs were greatest. The Legislature 
appropriated a significant increase in funding, now being realized in increased 
numbers of personnel and changes in local programs through training and family 
supports. The earlier, multiple divisions of early intervention based on population 
centers, geography, etc., have merged into one El program with two districts for the 
state of Nevada. 

Despite the fuzziness of their data, the Part C Committees persisted in looking at 
results. [WRRC Evaluation Report, 12/02] Their focus on the data, applying and using 
them have enabled NV Part C to reliably display and act on the strategies and 
activities in their improvement efforts. The agency holds regular “data checks” with 
stakeholders to verify the extent to which targets are being reached or need adjustment. 

“I think one of the smartest things we did was invoive faciiitators from the WRRC, peopte from 
outside ... abie to be objective. [They] expiained the RA modei and how and why it worked. 

We were dear enough we coutd exptain it to our Steering Committee; used it to work through 
indicators and measures. Reaiiy gave us a pian that wouid give us direction and data that 
woutd answer what we wanted to know. Our struggie ait atong has been our data. Continues 
to be something we work on. ” 

“We are reconvening the Steering Committee and iCC; this is a reai data coiiection point, six 
months after [the iP document was submitted]. Some Ei programs were impiemented quicker 
than others; this wiii be our first opportunity to took at the data to see what’s different, what 
resuits we can see. it’s our opportunity to re-examine and ensure that what we pianned is stiii 
vaiid. Key time to refiect and get a handie on what we need to be doing, where we need to be 
going next. Look at data: is it an isoiated probiem, or is everybody doing not so weti. Looking 
at those data again, suggestions for what we shoutd be doing in the next six months. ” 
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OSEP’s NV contact held a teleconference in March 2004 for a status check. In 
addition to the NV Part C staff, this forum included NECTAC and WRRC staff, 
providing input and perspective from all the partners in NV Part C’s improvement 
efforts. Key themes were the commitment to the process, the continued scrutiny and 
application of data, and using a consistent process throughout. Having TA providers 
give a nationai perspective iends invaiuabie credibiiity to the process and discussions. 
Being abie to say “This is the intent” has heiped participants change perceptions. 



Results to Date: 




In July 2003, Nevada Part C was moved from the Department of Human Services to 
the Department of Health. By July 2004, the agency had become the Bureau of Early 
Intervention Services and recorded these major achievements in that one year’s time: 

V a 61% increase in employees (from 157 to 253) 

V an 1 1 % increase in number of children served 

V a “steady decrease” by 62% of children waiting more than 45 days for ini- 
tial IFSP development (510 recorded in September 2003; 196 in March 2004) 

V an effective “services re-engineering:’’ changing the philosophy behind and 
approach to direct services of children and families, including a streamlined, 
coordinated statewide process for appropriately finding and screening children 

NV Part C’s APR (3/04) describes the variety of effort and results during SY 2003; its 
report to OSEP (3/04) contains information on achievements and challenges up 
through December, 2003; the Anniversary Achievements from the Bureau of Early 
Intervention Services (7/04) provides the most current picture of the substantial 
progress Nevada’s Part C agency, its stakeholders and its local programs have made 
in a concentrated timeline. The following charts display those key indicator data 
(compiled from the NV Part C APR, 3/04). 
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Direct assistance from the WRRC and partnered support between NECTAC and the 
WRRC to NV Part C have supported that agency in making significant and measured 
progress in improving services and results for children in Nevada. 



“This process is one of the more positive things I’ve worked on in disabilities; lots of others 
feel like wastes of time. On this [the SA and IP] we saw beginning and end, and it was set 
up so I believed it could really make a difference, that I helped to make a positive process.” 



“I think the TA we got from WRRC was extremely helpful: helped us think about things dif- 
ferently. For example, we’d had training with Friedman [RA], but hadn’t thought about ap- 
plying it to the IP process until Stephen and Jane suggested it. They made suggestions that 
helped us stay on track. They were so willing to preplan, understand issues for NV, debrief 
every meeting to help us focus on what’s ahead, what’s needed to move forward for next 
time.” 
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The following charts on three key indicators appear in NV Part C’s APR. 
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PERFORMANCE MEASURE 2 
States’ Use of Results Data 



Endorsing the importance of focusing on results, RED workgroup members chose a per- 
formance measure to describe the extent to which RRC Network efforts assisted states in 
applying results-based decision making: Percent of states regularly implementing, review- 
ing and revising improvement plans based on results data. 

Establishing a data collection protocol 

Describing progress on this measure used a protocol outlining how this ‘regular review 
based on data’ would be measured. As originally envisioned, this protocol was meant to 
address both level of RRC involvement, and the data-based approach of the state. As two 
RRCs began piloting the protocol, the need for revisions became evident. The structure of 
the protocol emphasized the level of RRC involvement, but did not seem to capture a true 
sense of states’ results focus. In addition, there was an evolution in OSEP’s vision for states 
that was not reflected in the original protocol design. 

The RED revisions to the protocol included trying to resolve the tension between collecting 
information on RRC assistance begun in the previous OSEP construct and that being 
provided under the newly articulated OSEP approach. An additional consideration was the 
gradual inclusion of Part C Lead Agencies in RRC technical assistance. Each RRC was at 
varying levels of involvement with its regional Part C systems. In the end, the revised 
protocol was designed to capture RRC assistance across old and new approaches, B and C 
agencies, and the states’ perspective on the improvement planning process in general 
(whether CIMP or APR driven). [See Appendix A for the final protocol.] 

Results 

RRC Perspective on Improvement Planning Process 

RRC staff members were asked to identify improvement planning activities in states. Using 
definitions standardized for this inquiry, an individual RRC staff would mark “yes” if the RRC 
assisted the state in any capacity wWn an improvement planning activity. Five of the six 
RRCs provided data for states in their regions. The amount of state representation ranged 
from 1 1 states to one state, resulting in data for 37 Part B agencies and 36 Part C agencies. 

The following definitions defined the type of RRC involvement with states for this question: 

• Planning: Work with state leadership to layout the process. Assist in timeline 
development. 

• Facilitation: Lead meetings, design the agenda, lead public forums 

• Development: Design formats, present frameworks, create procedures, 
writing/assistance to writing 
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• Presentations: Offer information about the overview of the process, deliver 
OSEP’s message (regional or individual state) 

The following tables, for Part B and Part C agencies, display weighted distributions of 
activities by RRC staff in any of several improvement efforts. Since RRC involvement 
often occurred in more than one activity, multiple responses were possible, and the 
weighting formula was designed to equalize the varied number of states’ data 
submitted by each RRC (see footnotes for more detail). 



PART B Type of Involvement (Yes/No to Each) 


Activity 


Y 


Planning 


Facilitation 


Deveiopment 


Presentations 


Self-Assessment 


91 .89%' 


88.24%'" 


91.18% 


79.41% 


73.53% 


Improvement Planning 


89.19 


87.88 


84.85 


72.73 


84.85 


Focused Monitoring 


72.97 


37.04 


33.33 


25.93 


92.59 


Data Analysis/Data 

Management 

Systems 


59.46 


31.82 


13.64 


31.82 


95.45 


BPR 


37.84 


14.29 


0.00 


14.29 


78.57 


APR 


78.38 


79.31 


37.93 


62.07 


86.21 


GSEG 


59.46 


50.00 


36.36 


31.82 


68.18 


SIG 


67.57 


40.00 


52.00 


40.00 


84.00 


Other: CADRE 


20.00 


N/A 


N/A 


N/A 


N/A 



PART C Type of Involvement (Yes/No to Each) 



Activity 


Y 


Planning 


Facilitation 


Development 


Presentations 


Self-Assessment 


48.65% 


88.89% 


88.89% 


83.33% 


66.67% 


Improvement Planning 


50.00 


88.89 


83.33 


55.56 


55.56 


Focused Monitoring 


58.33 


19.06 


14.29 


14.29 


90.48 


Data Analysis/Data 
Management Systems 


21.62 


50.00 


37.50 


62.50 


75.00 


BPR 


2.78 


100. 


0.00 


100.00 


0.00 


APR 


52.78 


36.84 


10.53 


15.79 


64.21 


GSEG 


50.00 


44.44 


27.78 


38.89 


88.99 


SIG 


38.89 


21.43 


14.29 


14.29 


92.86 


Other: Regional Part C 
Workgroup 


20.00 


0.00 


0.00 


0.00 


100.00 



'The involvement was calculated using the following formula: (# of states that received type of assistance/# of states that were 
represented in this particular survey)* 100. For example, if MSRRC only included data for 6 states, and 5 received a particular 
type of assistance, the “Yes” response =83%. This formula was used to account for the varied number of states’ data submitted 
by each RRC (1-11 states) and to weight the percentages accordingly. 

^The type of involvement was calculated using the following formula: (# of Yes responses to type of involvement/# of Yes 
responses to a particular activity) *100. For example, of the 5 MSRRC states that received assistance, MSRRC was involved 
with planning activities for 4 of them (80%) and facilitation activities for 2 of them (40%). 
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State’s Perspective on Improvement Planning Process 

State respondents were asked four questions about their experiences with the improvement 
planning process. 

RRC Involvement : The first question asked what worked well with, and what might 
have improved the involvement of, the RRC. 

There were several themes among the narratives states provided for this question: 

What worked well: 

The general support the states receive from the RRCs, 

The skill-based expertise of the RRCs (e.g., planning, facilitation and 
organization). 

The network opportunities provided by the RRCs through connections to 
other states. 

The RRCs are a readily available and useful resource. 

The RRC consultants provide up-to-date information, and 
The RRCs work at meeting the individual needs of the states. 

Specific examples/comments included: 

♦ Self-assessment models 

♦ RRC-facilitated communication across agencies 

♦ Helping states gather stakeholder input 

♦ RRC overview of the CIMP process for stakeholders 

♦ Regional conference calls/support 

♦ RRC staff content expertise and facilitation skills 

♦ Accommodations for TA depending on need of the state 

♦ Assistance with GSEG and SIG applications 

♦ Evaluation, performance indicator, and results accountability TA/training 

Possible Changes: 

More clearly define the role of the RRCs, 

States take advantage of the resources made available through the RRCs more 
often, 

80 Consistency of the improvement planning process TA, 

More RRC staff per geographical area. 



Effects on state decision-making : For the question on the extent to which improve- 
ment planning activities moved the state toward data-based decision making, the 
mean across RRCs was 6.4 on an 8-point scale (1-low, 8-high). 
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Process influences : State respondents were then asked to describe the as- 
pects of the improvement planning process most influential in moving their 
state in the direction of data-based decision making. 

Themes in response to this question included: 

Communication among stakeholders/agencies, 

Reviewing the quality of the data 
Information and support provided by the RRCs 
Monitoring process/efforts, 

Standard process for reviewing the data, 

Accountability for data and results. 

Overall quality of RRC assistance : This item, ranked on an 8-point scale, was 
an overall mean of 7.1 for all RRCs. 
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PERFORMANCE MEASURES 3 and 4 
Client Satisfaction 



Performance Measure 3 : Percent of clients reporting satisfaction on 
information requests. 

In order to assess the quality of the RRC Network’s Information Services function, 
selected clients were sent a three-item survey. The survey item of interest for the 
network accountability framework asked about the quality of information provided. 

The ways in which each RRC selected the clients to receive the survey varied as did 
the intervals at which the survey was sent. Aside from the three items included in the 
survey, there was no uniform method for soliciting the responses. Some RRCs 
included the items in an e-mail form, others printed post cards and sent the survey 
via postal mail. This item was ranked on a scale of 1 - 5 (1-low, 5-high). 



Result : Across the network respondents (n=126) ranked the utility 
of information with a mean score of 4.4, being of “high quality.” 

Performance Measure 4: Percent of TAA evaluations that indicate TA was of 
high quality. 

In order to assess the quality of TA activities each RRC agreed to a common rating 
scale (1-low, 8-high) and protocol for using it to solicit response from clients. The 
Network agreement was to use this scale at the end of a specific TA activity, asking 
clients to rate the quality (outcomes accomplished, relevant to the need, facilitated 
solving the problem). 

Additionally, following completion of a Technical Assistance Agreement (TAA), or 
periodically in the course of a TAA, state contacts were asked to rate the overall 
quality of the TA provided (timeliness, relevant to the need, facilitated solving the 
problem). 



Result: When aggregated across these two scales, respondents 
(n=479) ranked TA activities with a mean score of 7.0, of “high qual- 
ity.” 
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Lessons Learned 



The challenges to mounting a first effort to establish, collect and report common per- 
formance measures across a national Network were significant but necessary to the 
RED Workgroup and the RRC Network’s commitment to program accountability. 

This ambitious effort yielded numerous lessons, expected and serendipitous. Meas- 
ures, protocols and expectations underwent revisions as plans were put into place 
and experiences suggested modifications. 

The lessons that follow are a selection of those synthesized by the RED Workgroup 
members presenting the process and provide results in a session at the American 
Evaluation Association’s annual meeting in November, 2004. They are framed in 
statements of action that will contribute to the next stage in the RRC Program’s dem- 
onstration of its efforts and effects. 

Communication 

Clarify wording of Performance Measure (PM) questions 
Include representation from all interested entities in the redesign of instruments 
Build in periodic check-ins of a PM’s ability to collect appropriate data 
Communicate regarding performance measures throughout the process, between 
and among Centers 

Actions... 

Set expectations with clients regarding data collection 
Ensure that data collections are timely as well as collected at regular, agreed 
upon intervals 

Ensure that we are asking the right questions and collecting the right data 
Distinguish data as useful to Centers and State clients 

Sometimes let data guide the performance measure, rather than the performance 
measure guide the data 

Examine staffing patterns for evaluation across all Centers 

Collaboration 

Establish expectations and common agreements among Centers 
Finalize a written Network evaluation plan 
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APPENDIX A 



Performance Measure Protocols 



Protocol for Performance Measure 2: 

Measurement of Intermediate Outcomes Resulting from 

Comprehensive, Data-Based Planning Technical Assistance Activities 



State Date 

RRC Perspective on Improvement Plannine Process [To be answered by the RRC1: 

1 . Please indicate the level and type of involvement for each of the following improvement 
planning activities for Part B. 

♦ If the RRC assisted the state in any capacity with an improvement planning activity, 
please mark “yes” in the first column next to the activity. 

0 Indicate the type(s) of assistance (planning, facilitation, development, and/or 
presentations) by marking “yes” or “no” in each of the columns within that 
row. 

0 Please refer to the definitions below for clarification of the types of assistance. 

♦ If the RRC did not assist the state with an activity, please mark “no” and go to the next 
activity/row. 

Definitions 

• Planning: Work with state leadership to layout the process. Assist in timeline devel- 
opment. 

• Facilitation: Lead meetings, design the agenda, lead public forums 

• Development: Design formats, present frameworks, create procedures, writing/ 
assistance to writing 

• Presentations: Offer information about the overview of the process, deliver OSEP’s 
message (regional or individual state) 



PART B TYPE of Involvement (Yes/No to Each) 


Activity 


Y/N 


Planning 


Facilitation 


Development 


Presentations 


Self-Assessment 












Improvement Planning 












Focused Monitoring 












Data Analysis/Data 
Management Systems 












BPR 












APR 












GSEG 












SIG 












Other (Please list) 
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2. Using the same proeess as above, please indieate the level and type of 

involvement for eaeh of the following improvement planning aetivities for Part C. 



PART C TYPE of Involvement (Yes/No to Each) 


Activity 


Y/N 


Planning 


Facilitation 


Development 


Presentations 


Self-Assessment 












Improvement Planning 












Focused Monitoring 












Data Analysis/Data 
Management Systems 












BPR 












APR 












GSEG 












SIG 












Other (Please list) 













3. Describe your overall assessment of the quality of RRC involvement: Specifically, what worke 
well and what would you change if you were starting over? [Please provide qualitative data or 
stories.] 



4. To what extent has working on Self Assessment, Improvement Planning, CIMP, CIFMS, et 
moved your state toward data-based decision making? 

Not at all (1) (2) (3) (4) (5) (6) (7) (8) Great extent 



5. What aspects of the entire improvement planning process were most influential in moving your 
state in the direction of data-based decision making? 



6. Please rate the overall quality of RRC assistance in your improvement planning process. 

Poor (1) (2) (3) (4) (5) (6) (7) (8) Excellent 
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Protocol for Performance Measure 3: 

Percent of clients reporting satisfaction on information requests 



In order to assess the quality of the network’s Information Service function, selected clients 
were surveyed and asked to rate the quality of information provided using the following scale: 



Was the information sent to you: 






Very useful Somewhat useful 

5 4 3 


2 


Not useful 
1 



Protocol for Performance Measure 4: 

Percent of TAA evaluations that indicate TA was of high quality 

In order to assess the quality of TA activities, clients were asked to rate all TA events accord- 
ing to the following scales: 



Please rate the overall quality of the TA provided (timelines, relevant to the need, facili- 
tated solving the problem): 

8 7 6 5 4 3 2 1 

Excellent Poor 



Additionally, following completion of a TAA, or periodically in the course of a TAA, state 
contacts were asked to rate the activities along the following scale: 



Please rate the overall quality of this activity (outcomes accomplished, relevant to the 
need, facilitated solving the problem): 

8 7 6 5 4 3 2 1 

Excellent Poor 
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APPENDIX B 



RRC Network 
Evaluation Framework 



APPENDIX B 

DRAFT 

RRFC Network Evaluation Framework 
Context for this Framework 

The RRC program has a long and often unsatisfying history trying to account for its 
work. In the early 70s, the RRCs developed and maintained a detailed tracking 
system, maintained with direction and support from the Coordinating Office for RRCs 
and an accountability task force. This system was neither as useful nor efficient as 
hoped. It collected information on task FTE, and numbers of clients served, products 
produced and information responses, but the data were not comparable across 
Centers and not knowingly used to inform decisions within RRCs or for the program 
as a whole. This system was scrapped in 1977, when a new RRC program structure 
was launched. 

The next several iterations of data collection and reporting suffered from lack of a 
common vision. Absent a clearly-articulated set of expectations and an 
acknowledgement of slightly different foci from region to region, RRCs gathered in 
1988 and channeled their frustrations and efforts into the RRC Impact Assessment 
System (IAS), the first agreed-upon description of the mission, purpose, goals and 
measures for the program. In 1992 the MidSouth RRC made another edition, and 
the IAS continued to serve as a reference in individual Centers and in the Network 
(predominantly through Reporting, Evaluation and Documentation (RED) Workgroup 
efforts). The 1994/95 third party evaluation of the RRC program employed some of 
the IAS concepts in its design, and suggested common indicators of successful 
RRCs. 

In 1997 the Network made another attempt to establish a common foundation in its 
proposal for an impact study of the RRC Program that would establish performance 
measures, provide feedback on the program to date, and inform the design for the 
upcoming RFP. The proposal was grounded in the IAS concepts, in findings of the 
external evaluation, in the collective program experiences and in the belief that the 
Network actions and results ought to be based on data. The proposal was turned 
down by OSEP, at the time intending that RRC resources be targeted to help states 
move into the requirements of the newly-reauthorized IDEA. 

The 1997/98 RRC RFP, mirroring IDEA’S emphasis on data, explicitly aligned the 
workscope with state systemic changes to improve outcomes. In addition, the RFP’s 
requirement for a third party evaluation of the program offered an unprecedented 
opportunity for stakeholders (funding agency, grantees and clients) to provide input 
on common measures across the Network. Again the RED Workgroup contributed 
design, forms and procedures recommendations and suggestions on how the results 
could best inform the Network and OSEP. Although this third-party evaluation itself 
did not move the Network further toward measures or mechanisms for better 
demonstrating effects and impact, its stimulus coincided with other substantial shifts 
in the program: RRCs find themselves focused on increasingly common clients 
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and approaches across regions as states respond to the CIMP, and the demand for 
accountability and program improvement information continues to increase. OSEP 
needs more discrete data to justify programs, states are required to report according 
to specific performance goals and indicators, and there is a commensurate need for 
RRCs to be accountable. 

The following represents the RED Workgroup’s most recent attempt toward a 
common and consistent Network framework to help us talk about what we do in 
common, understandable terms, and employ tools that will help confirm how what we 
do makes a difference. It complements recent Network efforts to describe the future 
needs for RRC TA services, linking RRC, state and OSEP accountability measures, 
and offers an approach to develop a workable framework for common, results-based 
accountability that minimizes cost and maximizes utility. 

In addition to its broader purpose of facilitating results-based accountability, this 
framework is designed to respond to the third party evaluation recommendations to 
the RRFC Network: 

♦ Define common evaluation criteria and measures and look for 
inter-relationships between RRC activities and field impacts, rather than linear 
cause-effect chains. 

♦ Develop methods for assessing ‘intervening events’ that erode or undermine 
the impacts of the RRC efforts and determine how such assessments can lead 
to strategies that help the RRCs respond to these events strategically. 

♦ Reinvest evaluation ‘lessons’ in RRC improvement and strategic planning 
processes. 

♦ Ensure that linkages and work with the OSEP TA&D network and US 
Department of Education funded projects are systematically documented. 

♦ Engage clients in the evaluation plan and process in order to a) fully integrate 
evaluation into the TAA process; b) expect states to contribute effort and data 
to evaluate results of assistance; and c) include clients views on the reasons 
for success and failure of TA. 

The Results Accountability process developed by Mark Friedman was used as a 
starting point for developing the Evaluation Framework. Key points of Dr. Friedman's 
Results Accountability process are: 

1 . Start with ends, work backward to means. [What do we want? How will we 
recognize it? What will it take to get there?] 

2. Be clear and disciplined about language; use plain language, not exclusionary 
jargon. 

3. Use data (indicators and performance measures) to gauge success or failure 
against a baseline. 

4. Consider the story (or epidemiology) behind the baselines. 

7. Involve a broad set of partners. 

8. Develop an action strategy. 

(See http://raguide.org/ for a detailed description.) 
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These key points align with the RRFC Network's approach. They are results-based, 
focus on common language, involve partners, and describe the context, not just re- 
port the data. 

Parameters & Principles 

Drawing from the Results Accountability method, experience working as a network, 
and recommendations from the third party evaluation report, the following principles 
for the Evaluation Framework were developed. 

Reflects the RRFC mission 

The general purpose of the RRFC Network - deduced from individual RRC mission 
statements - is to work collaboratively to provide technical assistance to State Educa- 
tion Agencies and their partners to assist them in systemic improvement policies, pro- 
cedures and practices which will result in quality programs and services for children 
and youth with disabilities and their families. The RRFC Network strives to build rela- 
tionships within and among the regions of the country in order to link resources and 
information designed to improve results for children and youth with disabilities and 
their families. 

Demonstrates effort as well as effects 

RRFC technical assistance is provided primarily at the state level, sometimes making 
connections to student level outcomes challenging. Reporting effort as well as ef- 
fects provides an opportunity to describe the context into which RRFC technical as- 
sistance fits and its intended contribution to improving state systems. 

Describes the "story behind the baseline" 

The story is a description of forces at work behind the data trend(s). Understanding 
this sfoAy could assist in developing strategies for addressing intervening factors that 
undermine effort and effects. 

Recognizes multi-layered logic models or multiple pathways 

The range in TA activities and context in which these are conducted require a flexible 
approach to evaluation which includes collection of both quantitative and qualitative 
data. The Evaluation Framework provides a common core that fits the whole with 
flexibility for individual RRCs to add, expand, and/or adjust as needed. 
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Performance Measures 

The first step in developing the Network Evaluation Framework was to identify the 
RRFC Network clients. The direct clients for the RRFC Network are the Part B/Part C 
agency staff. This relationship in the context of the larger system is depicted below: 




With the clients identified, a list of RRFC Network efforts (what we do) and effects 
(how well we do it) was generated using the Results Accountability method. The table 
below describes the general approach to listing these efforts and effects. (A com- 
plete list can be found at the end of this document.) 



Quantity 


Quality 


What We Do 


How Well We Do It 


Effort 


How many clients served? 
How many activities con- 
ducted? 


What percent of clients report satisfaction 
with service? 

What percent of activities are reported to be 
of high quality? 


Is Anyone Better Off? 


Effect 


How many clients report 
RRC services are helpful? 


What percent of clients report RRC 
services are helpful? 



From the list of effort and effects, the following are the primary performance measures 
chosen for the Network Evaluation Framework: 

♦ Percent of Part B/C staff reporting RRFC TA contributed to improving their state system. 

♦ Percent of states increasing the number of students with disabilities fully participating 
in the state accountability system. 

♦ Percent of states reporting improvement on performance of students with disabilities. 

♦ Percent of TAA evaluations that indicate TA was of high quality. 
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Using these performance measures the RRFC Network will describe the quality of 
RRFC effort and effect. In addition, a list of common terms and definitions has been 
developed to ensure clear communication of achievement. This list of terms can be 
found at the end of this document. 

Data Collection and Reporting 

Below is a description of the process for collecting data on the performance meas- 
ures. Each Center will report this information biennially to the Federal Resource 
Center. The final report will be a stand alone document that describes RRFC Net- 
work contributions to the national picture presented. 

Performance Measure Data Source(s), Collection, Analysis, and Reporting 

1 . Percent of Part B/C staff Each Center will survey clients (SEA Director, Part C lead, 

reporting RRFC TA and/or Primary TAA contact) using a set of common items, 

contributed to improving their These items may be included in existing procedures/forms 
state system. or distributed as a separate piece. 

Each Center will report their aggregated information to 
these items biennially to the FRC. 

Potential survey items: 

1 . IRD workgroup Network Evaluation: How was the 
information used? and How did the information you 
received affect the issues you are addressing? 

2. Survey items (2-3) which address evidence of change: TA 
achieved outcomes which measurably benefit children and 
youth, or resulted in systemic impacts, benefits, and 
changes or... 

2. Percent of states increasing A RED group member or members will be assigned 

the number of students with responsibility for collecting the information and reporting 

disabilities fully participating in biennially to the FRC. The following are potential data 
the state accountability sys- sources: 

tern. 1. NCEO survey results: survey instrument and June 2001 

Report available on NCEO web site 
[http ://ed ucation . u m n . ed u/nceo] 

2. Biennial Performance Reports: required goals and 
indicator data on participation in and performance on 
assessment, drop outs, and graduation. Next BPRs due 
5/31/2002. 

3. RRFC State Similarities Database: includes data on 
graduation rate. Update schedule to be determined. 

4. 4. Consolidated Performance Reports: Title I reporting of 
number of students with disabilities participating in 
assessment. 

5. Report on Ed Indicators with focus on Title I available at 
http://www.ed.gov/offices/OUS/PES/esed/2000_indicators/ 

6. State web sites. 
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3. Percent of states A RED group member or members will be assigned 

reporting improvement on responsibility for collecting the information and 
performance of students reporting biennially to the FRC. The following are 
with disabilities. potential data sources: 

1. NCEO survey results: survey instrument and June 
2001 Report available on NCEO web site 
[http://education.umn.edu/nceo] 

2. Biennial Performance Reports: required goals and 
indicator data on participation in and performance 
on assessment, drop outs, and graduation. Next 
BPRsdue 5/31/2002. 

3. RRFC State Similarities Database: includes data 
on graduation rate. Update schedule to be 
determined. 

4. Consolidated Performance Reports: Title I 
reporting of number of students with disabilities for 
each proficiency level. 

5. Report on Ed Indicators with focus on Title I 
available at http://www.ed.gov/offices/OUS/PES/ 
esed/2000_indicators/ 

6. State web sites. 

4. Percent of TAA ♦ Each Center will survey clients (SEA Director, Part 

evaluations that indicate C lead, and/or Primary TAA contact) using a set of 
TA was of high quality. common items. 

♦ These items may be included in existing 
procedures forms or distributed as a separate 
piece. 

♦ Each Center will report their aggregated 
information to these items annually to the FRC. 

Potential survey items: 

1 . IRD workgroup Network Evaluation: Was the 
information request response useful (1-5 scale)? 

2. Consumer satisfaction items (2-3) which address 
quality TA: was TA targeted, accessible, worth the 
effort? 
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Full list of RRFC efforts and effects generated using the Results Accountability process. 





Quantity 


Quality 




What We Do 


How Well We Do It 


Effort 


Clients 
# Part B staff 


Clients 




# Part C staff 


% lEP/IFSP of need 




# Children with IFSPs/IEPs 


Ratio of SEA staff to RRC staff + 




# Families with children in SpEd 


consultants 




# Other stakeholders 


Cost per TA agreement 

Staff turn over (% employed 1 year or 




Activities 


more) 




# Consultations (onsite, phone, e-mail) 

# Written, substantive recommendations 


% fully trained staff 




# Reports prepared 


Activities 




# Information request responses 


% On time responses to information 




# Products disseminated 


requests 




# Meetings facilitated 


% Consultation repeat customers 




# TAAs created 


% SEA staff engaged in RRC activities 




# Trainings/conferences 


Products per month (requested, other) 




# Evaluations 


% of states per month with active 




# Web site hits 


engagement (on site, phone) 

Products disseminated % of total “market” 
% of positive evaluations for training and/or 
facilitation 

% of satisfaction with evaluations 
Rate of web site hits per month 



Is Anyone Better Off? 


Effect 


# students with disabilities included in ac- 

countability system 

# states with PGs & Is in place 

# states who use PGs & Is to improve per- 

formance 

# states with “integrated initiatives” 

# states in compliance (based on OSEP 

involvement) 

# Part B/C staff who report that RRC 

helped them in their jobs 


% students with disabilities fully 

participating in the accountability system 
% states with PGs & Is in place 
% states who use PGs & Is to improve 
performance 

% states with “integrated initiatives” 

% states in compliance (based on OSEP 
involvement) 

% Part B/C staff who report RRC helped 
them in their jobs 
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APPENDIX C 



Network Accountability 
Report Pian 



APPENDIX C 



RRC Network Accountability Report 
Purpose, Timelines and Structure 
Updated June 2004 

Purpose of the Network Accountability Report 

As accountability for student results has become the target for all federal programs, connecting the 
effects to improved child/student results has been a challenge for the RRC Network. Since January 
2001, the RRFC Reporting, Evaluation, and Documentation (RED) workgroup has been working on an 
accountability framework to describe effects of RRC activities using a set of common measures. The 
RRC Directors approved the measures and plan to collect and report the data and the RED group de- 
veloped procedures for collecting the data. 

The Network Accountability Report will provide data on each of these measures and observations 
about the process of constructing and implementing the Network accountability framework. Though 
each RRC submits progress reports on efforts and effects, this report will provide a picture of effects of 
the RRC program - something not captured in the individual Center reports. 

Structure of the Network Accountability Report 

Each RRC has agreed to collect and report data on a set of common measures. The following out- 
lines the measures and format for reporting the data collected. 

I. Performance Measure 1 : Percent of states where RRC intervened and positive trend lines are 
evident in one or more of the OSEP priority areas. 

Each RRC will provide a story describing its activities and effects in one state. These 2-3 page 
stories will then be assembled to provide a picture of the Network on performance measure 1 . 
[Note: This is a shift from the original plan to collect data that would result in a report of percent of 
states with positive trend lines.] 

The elements of this story are: 

• What the RRC did 

0 State context (including barriers, variables) 

0 Partners involved 

0 Immediate results 

• Progress to date - including state reported 618 data where appropriate 
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II. Performance Measure 2: Percent of states regularly implementing, reviewing and revising im- 
provement plans based on results data. 

Each RRC will use an agreed upon form to collect data on this performance measure. The data 
will then be analyzed and a percentage reported for the Network as a whole. Observations and/or 
clarifications will be included as appropriate. 

III. Performance Measure 3: Percent of clients reporting satisfaction on information requests. 

Each RRC will collect and report data from selected clients requesting information using the fol- 
lowing item: 

Was the information sent to you: 

Very useful Somewhat useful Not useful 

5 4 3 2 1 

The percentage of clients finding this assistance of high quality (useful) will be reported for the 
Network overall. 

IV. Performance Measure 4: Percent of TAA evaluations that indicate TA was of high quality. 

Each RRC will collect and report data using the following items: 

For all events: (asked of participants in the event) 

Please rate the overall quality of this activity (outcomes accomplished, relevant to the need, facili- 
tated solving the problem): 

8 7 6 5 4 3 2 1 

excellent poor 

For end of a TAA, or periodically in the course of a TAA: (asked of state contacts for the TAA) 

Please rate the overall quality of the TA provided (timeliness, relevant to the need, facilitated solv- 
ing the problem): 

8 7 6 5 4 3 2 1 

excellent poor 

The percentage of clients finding this assistance of high quality will be reported for the Network 
overall. 



Timelines 



June 30: 
July 20: 

August 10: 



data collected and submitted to designated RED group members 

data analyzed and draft of analysis for each performance measure reviewed by 

RED workgroup 

draft of full report delivered to the RRC Directors and OSEP project 
officer 
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